Pillarboxing correction

ABSTRACT

A display control device used to govern non-content regions in a display space, and selectively determine data for display in the non-content regions is described. The display control device can identify the non-content regions, and determine types of data that can be filled in the non-content regions. Once determined, the fill data can be presented in the non-content regions concurrently with an image frame.

TECHNICAL FIELD

This invention relates to methods and systems for displaying content ona display device.

BACKGROUND

Portable electronic media players, such as small portable digital musicand video players, have become increasingly popular with consumers.Music jukeboxes, such as the Apple iPod®, available from Apple Inc.,from Cupertino, Calif. allow consumers to store and play video and audiofiles of different formats. Other functionalities include the ability torecord audio sound (e.g., human voices) or capture video content (e.g.,movies). Due to their smallsize, portable players provide users theflexibility, versatility and convenience to replay stored video andaudio data at virtually anywhere and at anytime.

Video content can be formatted and optimized for a particular type ofdisplay. Some video content can be optimized in accordance with anaspect ratio of a display on which the content is to be displayed.Aspect ratio is the ratio between the width and the height of a displayused to play the video content. If the aspect ratio for which the videocontent is optimized matches the aspect ratio of the display device,then the entire display space of the display can be utilized fordisplaying the video content. Conversely, if the aspect ratio for whichthe video content is optimized differs from that of the display device,only a portion of the display space can be utilized for displaying thevideo content.

SUMMARY

Systems, methods, computer-readable mediums, user interfaces and otherimplementations are disclosed for governing non-content regions in adisplay space of an image frame, and selectively determine data fordisplay in the non-content regions so as to enhance viewing experienceof the image frame being displayed. The non-content regions can first beidentified in each frame. The identification process can includeidentifying dimensions and locations of the non-content regions thathave no active video content. The space occupied by non-content regionscan be determined after a display area is established. Theidentification process may be followed by a determination of types ofdata that can be filled in the non-content regions. The types of filldata can include, but are not limited to, data associated with movieenhancing effects, video related information, and user provided data. Adisplay control device can be used to facilitate the determination ofthe non-content regions and data to be fill therein.

In some implementations, a method of enhancing visual effects includes:identifying one or more regions of a display space being displayed in acurrent frame by a display device, each region being a non-contentregion; evaluating display device information selected from a group ofcontent displayed in the display space or property informationassociated with the display device; determining fill data to bedisplayed in the one or more regions of the display space based on theevaluated display content; and displaying the fill data in eachnon-content region.

In some implementations, a method of enhancing visual effects includes:defining a display space including a display area and a fill area, thedisplay area displaying a current frame of video content; identifyingone or more regions of the display area; determining color dataassociated with the one or more regions of the display area; andapplying the color data for display in the fill area.

In some implementations, a display control device includes: a fill areaidentification module for identifying one or more non-content regions ofa display space in an image frame; a display area identification modulefor identifying one or more content regions in the image frame; and afill data module for identifying fill data to be displayed in the one ormore non-content regions.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the invention will be apparent from thedescription and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1A illustrates a block diagram of an exemplary media player.

FIG. 1B illustrates a block diagram of an exemplary display controldevice.

FIG. 2 illustrates an exemplary display associated with the media playerof FIG. 1.

FIG. 3 illustrates an exemplary letterboxing format for displaying avideo image optimized for a 16:9 aspect ratio when displayed on a 4:3aspect ratio display.

FIG. 4 illustrates an exemplary pillarboxing format for displaying avideo image optimized for a 4:3 aspect ratio when displayed on a 16:9aspect ratio display.

FIG. 5 is an exemplary process for displaying content on a displaydevice.

FIG. 6 illustrates an exemplary process for determining fill regions.

FIG. 7 illustrates an exemplary display area, fill area and fill regionsassociated with a display device.

FIGS. 8A and 8B illustrate how a particular frame is affected before andafter fill data has been added.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION System Overview

FIG. 1A illustrates a block diagram of an exemplary media player 100. AMedia player as used herein refers to any software, or sets ofexecutable instructions, hardware, or combinations thereof that areoperable to visibly and/or audibly communicate media data to an end-userusing a visual/audio output device. Media data can include one or moreof, by way of example, video data, audio data, graphical data, textdata, image data, and the like.

In some implementations, the media player 100 can be or be included in apersonal computer, notebook computer, mobile phone, portable audio/videoplayer, personal digital assistant (PDA), video recorder, digitalcamera, or other handheld or portable consumer electronic devices. Forthe purpose of brevity and not by way of limitation, the media player100 will be described in the context of a portable computing devicededicated to processing audio soundtracks, still images, video clips ormovies.

Referring to FIG. 1, the media player 100 includes a display 102 fordisplaying information associated with media files to the user. Thedisplay 102 can be a liquid crystal display device, an EL displaydevice, an FED (Field Emission Display) display device or other type ofdevice. The display 102 also can include a display control device 118operable to enhance video data being displayed, as will be described infurther detail below. In the implementation shown, media player 100 alsoincludes a processor 104 (e.g., CPU or controller), a storage device 106(e.g., hard disk), an audio interface 108 (e.g., for interfacing withspeakers), and one or more user input devices 110.

The processor 104 controls the playing of a particular media file suchthat, for example, upon receiving a user selection of the media file,the processor 104 accesses the storage device 106 to retrieve theselected file. Because the storage device 106 may be stored with anoverwhelming number of media files, the access speed needed to retrievethe selected file can be relatively slow. Thus, to enhance the accesstime to the media files stored in the media player 100, a cache 112(e.g., random access memory “RAM” or read-only memory “ROM”) can beincorporated. Media files that are frequently played can be loaded intothe cache 112, while other media files can be kept in the storage device106. Alternatively, processor 104 can allow media files to be accessedusing a network connection from a remote location (e.g., local areanetwork, wireless network, Internet, intranet, etc). For example, a usercan wirelessly assess and select a media file stored in the storagedevice 106 of the media player 100 to play the selected media file.

In some implementations, one or more network interfaces 114 also can beimplemented to provide Ethernet connection to the Internet, or supply aconnection bridge to other media players or media content sources. Oneor more of these components is coupled to one or more buses 116 (e.g.,USB, FireWire, etc.).

Generally, the processor 104 serves to execute application software thatcontrols the operation of the media player 100. The processor 104performs basic tasks, including but not limited to: recognizing userinput from the input device 110; sending media files to the display 102and audio interface 108; keeping track of media files and directoriesstored in the storage device 106; controlling external peripheraldevices; and managing traffic on the one or more buses 116. Theprocessor 104 also can perform processing operations, including specialeffects processing associated with media files stored in the storagedevice, as will described in further detail below.

The storage device 106 can permanently or temporarily store video data,audio data and the like, and can provide high capacity storagecapability for the media player 100. Media files stored in the storagedevice 106 can be retrieved by the user input device 110.

The input device 110 allows a user of the media player 100 to controland interact with the media player 100. The input device 110 can take avariety of forms, such as, but is not limited to, a button, keypad,touchpad, dial or click-wheel.

In some implementations, the input device 110 is a graphical userinterface (GUI). In these implementations, the GUI provides a bridge tothe stored media files for a user of the media player 100. For example,the GUI can provide a computer generated image that simulates a controlpanel of a hardware media player 100, and includes information generallynot accessible in a hardware media player. For example, the computergenerated image can include an index of media files whose content can beretrieved and executed by the media player 100. The media player 100 canbe initialized when a play selection has been made by a user. When theuser desires to have the media player 100 play a particular media file,the index of media files can be invoked and displayed to the user on thedisplay 102 so that a selection can be made.

The index of media files can include an index of producers, artists,directors photographers, originators, etc. associated with the mediafiles. The index of media files can be generated by the media player100, for example, by reading entries of one or more directories of thestorage device 106 or cache 112. Other organizational structures can beincluded (e.g., favorites) The media player 100 can search all suchdirectories, and retrieve only media files having a file extensioncompatible with the media player 100. The retrieved media files are thendisplayed in the form of an index on the display 102 to prompt for userselection. If desired, the GUI can further limit the number of mediafiles displayed on the display 102 to selected ones of the media filesassociated with a particular producer, artist, photographer, director,originator, etc.

A user can browse through the index of media files, and when the userdesires to play a particular media file listed in the index, the GUIallows the user to simply navigate and select this media file from theindex using the user input device 110.

In some implementations, the GUI can include a toggling function forswitching between a window that displays the index of media files and awindow that displays status information associated with a particularmedia file (e.g., time duration of a media file) that is selected forplay by the media player 100.

In another implementations, the GUI can include a user-defined playlist.A playlist can be created by a user, and selected one(s) of the mediafiles can be added to the playlist through the GUI. Media files definedin the playlist can be played without constant user interaction.

During operation, a user selects a desired media file to be played bythe media player. Upon making and receiving the selected media file, theprocessor 104 accesses the storage device 106 or the cache 112 ornetwork to locate the selected media file. The processor 104 retrievesthe selected media file from a location in which it is stored, and playsthe media file. The media file can be played almost substantiallyimmediately following the user's selection. In some implementations, ifthe media file contains audio data, an encoder/decoder can beincorporated into the media player 100 to produce audio signals foraudio speakers (e.g., headphones) coupled to the audio interface 108.The speakers can either be internal to the media player 100 or beprovided as a separate unit to the media player 100.

Aspect Ratio Overview

As discussed previously, the media player 100 can store and play mediafiles stored in the storage device 106 or from other locations. In someimplementations, media files include video content, such as, but is notlimited to, video clips and movies. Generally, video content includesone or more consecutive frames of video and/or image data, and can bedisplayed to a user of the media player 100 on a display 102. FIG. 2illustrates an exemplary display 200.

Referring to FIG. 2, the display 200 includes a display space 202 fordisplaying video content 206. The display 200 also defines a specificdisplay area 204 for displaying the executed video content 206.

As discussed above, video content can be formatted and optimized in amanner that suits a particular type of display used to display the videocontent. For example, video content containing video and audio data canbe formatted and optimized based on an aspect ratio of the display. Theaspect ratio defines the relationship between a dimensional width and adimensional height (or horizontal dimension and vertical dimension) ofthe display area 204 in which video content 206 is to be displayed.Conventionally, video content can be formatted according to a 4:3 aspectratio, also commonly known as the “full screen” ratio. In the 4:3 aspectratio, the display height is 75% less than the display width of thevideo content (i.e., 1.33 times greater in width than height).Alternatively, video content can be formatted according to a 16:9 aspectratio, also commonly known as the “widescreen” ratio. In the 16:9 aspectratio, the display width of the video content is 1.78 times greater thanthe display height. Video displays such as televisions generally have anaspect ratio of 4:3 or 16:9.

When the video content is produced, the type and the aspect ratio of adisplay being used to display the video content may not be immediatelyknown. Even if this information is known, the video content generallycan only be optimized to suit one type of display (e.g., 4:3 aspectratio display or 16:9 aspect ratio display). In order to display videocontent of a certain aspect ratio on a video display with anincompatible aspect ratio, various methods including letterboxing andpillarboxing techniques are used to provide such a capability, as willbe described in further detail below.

Letterboxing Overview

A typical framework for a display space generally includes a displayarea for displaying image frames and a non-content area. Image framescollectively represent a number of scenes, and each image frame cancontain one or more graphical images. Image frames are ordered accordingto the temporal position of the scenes they represent in a videosequence. By taking a rapid sequence of the frames and displaying theframes at a same rate at which they were recorded, the motion that formsthe basis of the image frames can be produced.

To accommodate an image frame having an aspect ratio higher than theaspect ratio of a display, a letterboxing format can be used. In theletterboxing format, a high aspect ratio image frame is displayed acrossthe full width of a lower aspect ratio video display. Because the imageframe occupies less than the available display area of the display,portions of the display space (namely, the non-content area) of thedisplay are blank. The blank areas can be covered with black bands, aswill be explained in reference to FIG. 3, to provide a theatrical effectof the video content.

FIG. 3 illustrates a letterboxing format for displaying an image frameoptimized for a 16:9 aspect ratio display on a 4:3 aspect ratio display.

Referring to FIG. 3, a 4:3 aspect ratio display 300 is shown. An imageframe 306 optimized for a 16:9 aspect ratio display is visuallydisplayed in a display area 304 of the display 300. As shown, while thedimensional width 314 of the display 300 is substantially the same asthe dimensional width 316 of the display area 304, the dimensionalheight 312 of the display area 304 is shorter than the dimensionalheight 310 of the display 300.

Due to a difference in aspect ratio between the video image 306 and thevideo display 300, the video image 306 cannot fully occupy every pixelavailable in the display space 302 available in the display 300. Asshown, regions 308 located at the top and the bottom of the displayspace 302 are vacated, and a pair of horizontal bands, called mattes,are placed over these non-content regions. The thickness of the mattesmay depend on the display aspect ratio intended by the image frame 306.For example, an image frame optimized for a 1.85:1 aspect ratio displaybut shown using a 4:3 aspect ratio display has thinner mattes than animage frame optimized for a 2.4:1 aspect ratio display but shown using asame display (e.g., 72% display area and 28% unused regions in theformer compared to 56% display area and 44% unused regions in thelatter).

Pillarboxing Overview

To accommodate an image frame optimized for a particular display havingan aspect ratio lower than the aspect ratio of an actual display used todisplay the image frame, a pillarboxing format can be used. FIG. 4illustrates a pillarboxing format for displaying an image frameoptimized for a 4:3 aspect ratio display on a 16:9 aspect ratio display.

Referring to FIG. 4, an image frame 402 optimized for a 4:3 aspect ratiodisplay is shown in a display area 404 of a 16:9 aspect ratio display400. In the pillarboxing format, the dimensional height 408 of thedisplay area 404 is the same as the dimensional height 410 of thedisplay space 406, while the dimensional width 412 of the display area404 is narrower than the dimensional width 414 of the display space 406.

Due to a difference in aspect ratio between the image frame 402 and thedisplay 400, portions of the display area 404 are not used duringplayback. These portions are regions 416, which are non-content areasthat remain blank during playback of the image frame 402.

To remove blank regions 416, conventional techniques including zoomingand panning are used. Zooming allows an image frame to be zoomed to filla display space in one or more dimensions. For example, in thepillarboxing format shown in FIG. 4, the pillarbox areas 416 arecropped, and the image frame 402 is magnified to fill the 16:9 aspectratio display 400 (or in the letterboxing format shown in FIG. 3, themattes 308 are cropped, and the image frame 306 is expanded to fill the4:3 aspect ratio display 300). The image frame is stretched whenhorizontal zooming and vertical zooming are performed.

In this method, the dimensional width 412 of the image frame 402 isexpanded horizontally to align with the edges of the 16:9 aspect ratiodisplay 400. While this zooming technique may efficiently increase thesize of image frame 402, the magnification also causes verticaldimensions of the image frame 402 to be expanded. Particularly, portionsof the image frame 402 at the top and bottom of the display space 406extend past the physical limit of the display 400 during zooming. As aresult, similar to the blank regions 416, portions of the image frame402 also are cropped during playback.

The conventional panning technique also has similar drawbacks. Referringto FIG. 3 as an example, in a panning operation, the dimensional height312 of the image frame 306 is expanded vertically to align with the topand bottom edges of the 4:3 aspect ratio display 300. However, whileportions of the image frame 306 displayed proximate to the top andbottom of the video display 300 are preserved, this expansion causes theperiphery regions of the image frame 306 to extend past the physicallimits of the display 300, resulting in a loss of image data near theperiphery regions of the display space 302.

In addition to a loss of image data, the aforementioned techniquesdistort the quality of the image frame, for example, by compressing orstretching the image frame to fit the entire display area. Thisadversely affects the resolution of the image frame when shown on thedisplay.

Display Control Device

Accordingly in some implementations, a display control device is used togovern the non-content (i.e., non-image) regions (e.g., the mattes) in adisplay space, and selectively determine data for display in thenon-content regions so as to enhance viewing experience of the imageframe being displayed.

FIG. 2 illustrates exemplary components of a display control device.Referring to FIG. 2, a display control device 118 includes a fill datamodule 120, fill area identification module 122, fill region module 124,display area identification module 126, preview module 128 andconfidence indicator module 130. The display control device 118 can bean integrated component of a media player, or alternatively, the displaycontrol device 118 can be formed as a module separate from the mediaplayer. In some implementations, the fill area identification module 122of the display control device 118 can be configured to identify thenon-content regions (e.g., mattes) in each frame. The identificationprocess can include identifying dimensions and locations (i.e.,coordinates) of the non-content regions that have no active videocontent. The space occupied by non-content regions can be determinedafter a display area is established The display area identificationmodule 126 of the display control device 118 can function to identifythis display area (i.e., content regions) used for displaying imageframes.

The identification process may be followed by a determination of typesof data that can be filled (hereinafter “fill data”) in the non-contentregions using the fill data module 120. The types of fill data caninclude, but are not limited to, data associated with movie enhancingeffects (e.g., color information), video related information (e.g., textinformation such as caption, title, chapter, review, synopsis, agerating), and user provided data (e.g., clock, stock ticker, weatherinformation and phone call identification if the display device supportsvideo conferencing).

Once determined, the fill data can be presented in the non-contentregions concurrently with the image frame. In some implementations, thefill data can accommodate frame or scene changes (e.g., as the action ora specific character moves in the image) to ensure that the fill dataremains consistent and coherent with the image frame being displayed inthe display area. In these implementations, fill data can be evaluatedand re-evaluated in each frame to effectuate temporal consistency.Previous and subsequent frames also can be considered for enhancing filldata in a current frame.

A process 500 for presenting media is exemplarily shown in FIG. 5.Referring to FIG. 5, at step 502, property information associated with adisplay used to display the image frames is identified. The propertyinformation can include, but is not limited to, information associatedwith aspect ratio, dimensions of a display space and the like. At step503, concurrently or sequentially, information associated with the imageframes to be shown by the display including, for example, aspect ratio,language, caption or the like, is also received or identified.

Based on the identified display property information and contentinformation, the aspect ratio of both the display and the image framescan be determined in step 504. Based on the determined aspect ratios,one or more non content display areas are identified in step 505. Theprocess of identifying non content display areas can include computingthe dimensional height and width (i.e., the size) of each non-contentregion. In some implementations, if the dimensional height or width of anon-content region is less than a minimum threshold, it can be assumedthat either the available display space of the display is fullyutilized, or that the size of the non-content region is too small to befilled with fill data that can render a sufficient visual effect. Inthis case, process 500 can be terminated in the current frame, andreinitiated for a next frame. Alternatively, if the dimensional heightor width of a non-content region meets the minimum threshold, thephysical size (e.g., number of pixels) of the non-content region to befilled with fill data (hereinafter “fill area”) is determined.

At step 506, fill data for the non-content areas is identified. Filldata can include, but is not limited to, text information, color dataand the like. In implementations where fill data includes color data,the color data can be enhanced to supplement realism and visual cues ofthe image frame being displayed, as will be described in further detailbelow.

FIG. 6 illustrates an exemplary process 600 for filling color data, andFIG. 7 exemplarily illustrates a display area, fill area and fillregions.

Referring to FIGS. 6 and 7, at step 600, a fill area 704 isautomatically identified. The fill area 704 can be identified by adisplay control device based on display property information, contentinformation or display format compatibility. For example, if it isdetected that image frames are displayed in a letterboxing format,mattes such as those shown in FIG. 3 can be identified as fill area.

In some implementations, fill data is determined based on how well thedisplay area and the fill area can be graphically merged to reduce thedegree of color dissimilarity between their background color, or tominimize noticeable artifacts that may occur when they are merged. Forexample, if the mattes 308 shown in FIG. 3 are identified as fill areas,then fill data that can diminish the appearance of the black bands, oralternatively replace the color of the black bands, can be empiricallydetermined.

Generally, color to be included as the fill data can be described interms of hue (chromatic color), saturation (color purity) and brightness(light intensity). By adjusting hue, saturation and brightness invarious ways, a combination of lighting effects such as, highlightededges, shaded areas and shadows, can be added to the fill area tosupplement the image frames shown in the display area. Thus, in someimplementations, if it is desired that the color data to be filled in afill area should, for example, smoothly blend the fill area into thebackground color shown in a display area, various degrees of hue,saturation and brightness can be employed (e.g., using RGB or CYMKmodel) for determining the color data.

For example, if the background color of an image frame being displayedis similar to that exhibited by a blue ocean, the display control deviceretrieves color data associated with the blue ocean (e.g., from filldata module 120). The retrieved color data can then be input into thefill area so as to seamlessly blend the fill area into the display areawithout introducing any sharp visual artifacts.

However, due to frame transition and associated changes in lightillumination, fill data may not fully alleviate the problem of colordiscontinuity or sharp transition that occurs at the boundary (see 710of FIG. 7) that divides the display area and the fill area.Frame-by-frame gradient operations also can give rise to strong temporalartifacts which can be noticeable during playback of the video content.

Accordingly, in some implementations, in addition to placingconsiderations on similarity in color between the display area and thefill area, the display control device can refine the fill data tosignificantly minimize undesired hard edges around the boundary oralternatively consider more than one frame (either past of futureframes) when making fill data decisions. For example, the displaycontrol device can specify a particular tint and/or shade to the filldata so that the resultant colors include different colors of the samehue with varying saturation and brightness. This can be used inconjunction with color intensity and lighting variation to enhance theoverall blending effect of the fill area.

In implementations in which the image frame exhibits a gradientbackground color, the display control device can analyze contributionsof the gradients across the display area, to calculate an average color(e.g., average color histogram) exhibited by the gradient backgroundcolor. The display control device can subsequently implement the averagecolor as the fill data and perform local operations to further refinethe visual effect of the fill data The visual effect can resemble agradient texture or a texture with fading colors. For example, agradient texture can simulate the fading of a color that begins at oneedge of the fill area and ends at the opposite edge of the fill area.

In some implementations, the fill region module 124 can divide the fillarea into a plurality of contiguous regions, and each region isevaluated separately to determine fill data. Again to alleviate hardedges or transitions, adjacent regions can be considered when makingfill data decisions.

In some implementations, for each region of the fill area an averagecolor is determined to individually accommodate different colortransition points. If desired, the display area also can be segmentedinto one or more display regions. For example, to compute the averagecolor, a color histogram associated with each display region isevaluated. Generally, each display region includes one or more pixelsand each pixel displays can have an associated color or shading (e.g.,grayscale). A color histogram that is representative of the colorsexhibited by the pixels in a corresponding display region can beobtained by categorizing the pixels into a number of bins based on colorand counting the number of the pixels in each bin. Based on the colorhistogram obtained from each display region, an average color histogramfor a particular image frame can be calculated by, for example, summingthe color histogram of each display region in the frame, and dividingthe summed histogram by the number of display regions in the frame.

Although the above technique is described in relation to the regionsadjacent to the boundary between the fill area and the display area, theentire display area also can be used in determining the average colorexhibited by the gradient background color.

As described above, the fill area can be segmented into one or more fillregions 706 (step 604) to, for example, individually accommodate thecolor associated with the one or more color transition points. Fillregions 706 can overlap with other fill regions, and the fill regions706 can collectively cover the entire or portions of the fill area 704.Each fill region 706 can be evaluated to determine fill data for arespective region (step 606). For example, color data can be determinedin a manner similar to those described above (e.g., to reduce the degreeof color dissimilarity between color of the fill area and backgroundcolor of the image frame).

For example, if a gradient background color exhibited in a particularimage frame includes a color transition from white (e.g., occupies theleft region of a display area) to black (e.g., occupies the right regionof a display area), the fill area can be segmented into two or more fillregions (e.g., a left fill region and a right fill region) each beingassigned color data that correspond to one of the background colors(e.g., the left fill region includes light color data associated withwhite and the right fill region includes dark color data associated withblack).

At step 608, the fill regions 706 are filled with fill data determinedat step 606.

If desired, the fill data can be monitored on a continuous and ongoingbasis for adaptively responding to frame/scene transition and the like.For example, fill regions can be re-segmented to accommodate differentbackground colors exhibited in different image frames.

FIGS. 8A and 8B illustrate how a particular frame is affected before andafter fill data has been added. Referring to FIG. 8A, a preview message800 is shown. The preview message 800 is graphically displayed in aletterboxing format, and a pair of black horizontal mattes 802 a and 802b are placed over and under the preview message 800. Using one or moreof aforementioned techniques (e.g., average color histogram), fill datais determined (e.g., the background color of the preview message 800 iscomputed), and used to fill the horizontal mattes 802 a and 802 b withdata that corresponds with the background color (e.g., green). Theresult is shown in FIG. 8B, in which the black horizontal mattes 802 aand 802 b are completely removed after fill data has been filledtherein.

In some implementations, the fill data can be refined by using atemporal algorithm that utilizes one or more previous or subsequentimages frames. For example, a set of color pixels can be collected froma first frame and a subsequent frame to determine a luminance differencebetween the frames. Responsive to the determined luminance difference, asuitable color data to be filled in the fill area for both frames isdetermined. Other color statistic, such as, but is not limited to, meanvariance and entropy for each frame or differences of these valuesbetween pairs of frames also can be employed. The color data in the fillarea can subsequently be adjusted for successive frames.

In another implementations, rather than using only a subset of previousand successive frames, all frames of a scene are considered whenevaluating color contributions from a subset of frames and determining asuitable fill data (e.g., color) to be applied to the fill area of acurrent frame. For example, the process can begin with determining, foreach frame, a representative value indicating the average color for thecorresponding frame. The representative value for each frame is thencompared and analyzed. A global or median value can be derived based onthe comparison. The median value can be a value temporally medial to afirst frame and a next frame that can be used in determining data (e.g.,a color) that corresponds to the median value for the current frame.

If desired, a preview algorithm also can be performed using the previewmodule 128 so that a user can evaluate the effect of the fill data priorto being implemented in the fill area. The preview control can beimplemented as a thumbnail or icon displayed along a side of a displayarea, or anywhere on the display space. The displayed thumbnail or iconcan be transparent or semi-transparent, and the display thereof can beprovided in a separate preview window. To initiate the preview control,a user can enter a preview command to a processor of a media playerthrough a user input interface or input device to cause the media playerto generate a list of one or more suitable colors. The user can navigateand select a color of interest for instant implementation.

In some implementations, the display control device may optionallyinclude a “confidence” indicator module 130 that provides a confidencevalue indicator to analyze whether the visual effect expressed by theresultant fill data should be toned down or brightened up. Theconfidence value indicator can be assigned a confidence value on asliding scale such as 1 to 5 (where 1 represents the highest degree ofcolor similarity) based on the closeness of the color matching betweenbackground color of the image frame and that represented by the filldata. The confidence value can be determined based on a ratio between amedian color associated with the image and the color from all pixels inthe current frame. Alternatively, the confidence value can be determinedbased on the ratio between a median color associated with the image andcolors from pixels that lie on the boundary that divides the displayarea and the fill area. Once a confidence value is obtained, the filldata can be enhanced to either, for example, increase or decrease itssaturation based on the confidence value.

For example, if a particular frame depicts a bomb explosion, whichgenerates mostly a white flash across the image frame, the fill data caninclude a substantially white color. In this case, the confidence numbercan be a high value, enabling the display control device to completelyremove and replace the mattes with a solid and bright white color.

The confidence number also can be based on a ratio of the backgroundcolor to all other colors in the frame. For example, the backgroundcolor shown in the preview message of FIG. 8A is green and the color ofthe text is black. The ratio of green color that occupies the frame ishigher than that occupied by black color. Accordingly, the confidencenumber can be assigned a high value in this example, enabling thedisplay control device to completely remove and replace the mattes witha solid and bright green color.

Conversely, in a frame that depicts a human crowd, no singular color isdominant. Thus, the confidence number can be assigned a low number,indicating that the display control device can optionally refine thefill data to reflect a toned-down effect.

In some implementations, the user may specify to the display controldevice a confidence threshold. When a confidence threshold is specified,a confidence number with a value equal to or lower than the confidencethreshold would require the display control device to re-evaluate thebackground of the particular frame so as to adjust the overall effect ofthe fill data.

If suitable color data is found, a verification refinement process canbe established to assure that the color data does not create effects ofsharp transitions, hard edges or noticeable artifacts, unless theprocess has obtained a high confidence value suggesting that the effectsare desirable.

The invention and all of the functional operations described herein canbe implemented in digital electronic circuitry, or in computer hardware,firmware, software, or in combinations of them. The invention can beimplemented as a computer program product, i.e., a computer programtangibly embodied in an information carrier, e.g., in a machine readablestorage device or in a propagated signal, for execution by, or tocontrol the operation of, data processing apparatus, e.g., aprogrammable processor, a computer, or multiple computers. A computerprogram can be written in any form of programming language, includingcompiled or interpreted languages, and it can be deployed in any form,including as a stand alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment. Acomputer program can be deployed to be executed on one computer or onmultiple computers at one site or distributed across multiple sites andinterconnected by a communication network. Apparatus of the inventioncan be implemented in a computer program product tangibly embodied in amachine-readable storage device for execution by a programmableprocessor; and method steps of the invention can be performed by aprogrammable processor executing a program of instructions to performfunctions of the invention by operating on input data and generatingoutput.

The invention can be implemented advantageously in one or more computerprograms that are executable on a programmable system including at leastone programmable processor coupled to receive data and instructionsfrom, and to transmit data and instructions to, a data storage system,at least one input device, and at least one output device. Each computerprogram can be implemented in a high-level procedural or object-orientedprogramming language, or in assembly or machine language if desired; andin any case, the language can be a compiled or interpreted language.

Suitable processors include, by way of example, both general and specialpurpose microprocessors. Generally, a processor will receiveinstructions and data from a read-only memory and/or a random accessmemory. Generally, a computer will include one or more mass storagedevices for storing data files; such devices include magnetic disks,such as internal hard disks and removable disks; a magneto-opticaldisks; and optical disks. Storage devices suitable for tangiblyembodying computer program instructions and data include all forms ofnon-volatile memory, including by way of example semiconductor memorydevices, such as EPROM, EEPROM, and flash memory devices; magnetic diskssuch as internal hard disks and removable disks; magneto-optical disks;and CD-ROM disks. Any of the foregoing can be supplemented by, orincorporated in, ASICs (application-specific integrated circuits).

A number of embodiment of the invention have been described.Nevertheless, it will be understood that various modifications may bemade without departing from the spirit and scope of the invention.Accordingly, other embodiments are within the scope of the followingclaims.

1. A method comprising: identifying one or more regions of a display space being displayed in a current frame by a display device, each region being a non-content region; evaluating display device information selected from a group of content displayed in the display space or property information associated with the display device; determining fill data to be displayed in the one or more regions of the display space based on the evaluated display content; and displaying the fill data in each non-content region.
 2. The method of claim 1, wherein property information includes information associated with one of an aspect ratio of the display device or dimensions of the display space.
 3. The method of claim 1, wherein identifying one or more regions includes determining a dimensional width and a dimensional height of each non-content region, and aborting display of the fill data if one of a dimensional height or width does not meet a predetermined threshold.
 4. A method comprising: defining a display space including a display area and a fill area, the display area displaying a current frame of video content; identifying one or more regions of the display area; determining color data associated with the one or more regions of the display area; and applying the color data for display in the fill area.
 5. The method of claim 4, where determining color data includes deriving an average and median color data value for each of the one or more regions.
 6. The method of claim 5, where applying the color data includes correlating the average color data value with the median color data value to produce a suitable color for display in the fill area.
 7. The method of claim 4, where determining color data includes considering a background color of the current frame being displayed in the display area to enhance the color data.
 8. The method of claim 4, further comprising refining the color data to ensure color consistency between color data associated with the one or more regions of the display area and color data applied in the fill area.
 9. The method of claim 8, where refining the color data includes de-emphasizing the color data to create a toned-down effect.
 10. The method of claim 4, further comprising refining the color data including applying a temporal algorithm to the color data, the temporal algorithm averaging a color texture associated with the current frame, a previous frame and a next frame, and displaying the averaged color texture in the fill area.
 11. The method of claim 4, further comprising refining the color data including blending determined color data from the one or more regions with color data associated with those displayed in the fill area to produce a gradient.
 12. The method of claim 4, further comprising verifying the color data prior to applying the color data in the fill area to ensure color consistency in relation to the color data associated with the one or more regions of the display area.
 13. The method of claim 4, further comprising refining the color data including collecting color data from a previous frame and a subsequent frame of the video content.
 14. The method of claim 13, where collecting color data includes determining a luminance difference between the previous frame, the current frame and the subsequent frame.
 15. The method of claim 4, further comprising refining the color data including collecting color data from all image frames of the video content for evaluating color distribution to be applied in the current frame.
 16. The method of claim 15, further comprising: determining a representative value indicating an average color for each frame; and comparing the representative value for each frame to derive a global value.
 17. The method of claim 16, where the global value is a value temporally medial to a previous frame and a subsequent frame to be used in determining the color data for the current frame.
 18. A display control device, comprising: a fill area identification module for identifying one or more non-content regions of a display space in an image frame; a display area identification module for identifying one or more content regions in the image frame; and a fill data module for identifying fill data to be displayed in the one or more non-content regions.
 19. The display control device of claim 18, further comprising a preview module for previewing a visual effect of the fill data prior to being displayed in the one or more non-content regions.
 20. The display control device of claim 18, wherein the visual effect of the fill data is previewed in a thumbnail or icon provided along a side the image frame.
 21. The display control device of claim 18, further comprising a confidence indicator module for assigning a confidence value for analyzing closeness of a color presented in a background of at least one of the one or more content regions and that represented by the fill data.
 22. The display control device of claim 21, wherein the fill data is re-evaluated if the confidence value is lower than a predetermined confidence threshold.
 23. The display control device of claim 22, wherein the predetermined confidence threshold is provided by a user.
 24. The display control device of claim 18, wherein the fill data includes one of data associated with movie enhancing effects, video related information, or user provided data.
 25. The display control device of claim 18, wherein the fill data is re-evaluated in each subsequent image frame.
 26. The display control device of claim 18, wherein the fill data is evaluated based on fill data applied to previous and subsequent image frames.
 27. The display control device of claim 18, further comprising a fill region module for dividing the one or more non-content regions into a plurality of fill regions each being evaluated separately to determine the fill data.
 28. The display control device of claim 27, wherein an average color is determined in each fill region, and the average color is used to produce fill data for corresponding fill region.
 29. The display control device of claim 18, wherein the fill data is monitored on a continuous basis for adaptively responding to transition to another image frame.
 30. A computer-readable medium having stored thereon instructions, which, when executed by a processor, causes the processor to perform the operations of: identifying one or more regions of a display space being displayed in a current frame by a display device, each region being a non-content region; evaluating display device information selected from a group of content displayed in the display space or property information associated with the display device; determining fill data to be displayed in the one or more regions of the display space based on the evaluated display content; and displaying the fill data in each non-content region.
 31. A computer-readable medium having stored thereon instructions, which, when executed by a processor, causes the processor to perform the operations of: defining a display space including a display area and a fill area, the display area displaying a current frame of video content; identifying one or more regions of the display area; determining color data associated with the one or more regions of the display area; and applying the color data for display in the fill area. 